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VALIDITY  OF  ASSOCIATE  RATINGS  OF  PERFORMANCE  POTENTIAL  BY  ARMY  AVIATORS 


BACKGROUND 

In  response  to  a  TRADOC  request,  the  Fort  Rucker  Field  Unit  of  the  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  has  undertaken 
research  to  determine  attributes  which  predict  aviators  who  are  potentially 
outstanding  combat  performers.  The  effort  consists  of  the  following 
three  interrelated  tasks:  (1)  Development  of  an  attack-pilot  profile  from 
analysis  of  proven  performance  (Eastman,  Leger  and  Shipley,  1977); 

(2)  development  of  a  rating  form  for  assessment  of  potential  attack  pilots; 
and  (3)  selection  and  evaluation  of  AH-1  trainees  using  the  findings  of 
tasks  1  and  2. 

Currently,  no  systematic  selection  of  candidates  for  AH-1  transition 
training  exists.  Many  trainees  are  assigned  to  transition  training  be¬ 
cause  they  are  due  for  reassignment.  A  need  exists  to  provide  unit 
commanders  with  reliable  and  valid  instruments  to  select  aviators  for 
AH-1  transition  training.  If  unit  commanders  had  more  and  better  infor¬ 
mation,  an  improved  fit  of  aviators  to  training  assignments  would  result. 
The  research  reported  herein  is  part  of  task  2  and  was  conducted  to 
determine  the  predictive  validity  of  unit  level  ratings  of  AH-1  candidates. 


OBJECTIVES 

The  principal  objective  of  this  research  is  to  determine  the  validity  of 
the  AH-1  candidate  evaluation  form  as  a  predictor  of  trainee  performance 
in  the  AH-1  transition  course. 


It  was  hypothesized  that  AH-1  (COBRA)  qualified  pilots  in  FORSCOM  units 
would  be  able  to  predict,  by  means  of  assoicate  ratings,  the  AH-1  training 
performance  of  aviators  from  their  units.  It  has  already  been  shown 
that  COBRA  pilots  in  cavalry  and  attack  units  demonstrate  high  inter-rater 
reliability  when  evaluating  the  potential  success  aviators  in  their  units  for 
AH-1  transition  and  gunship  pilot  duties  (Eastman  and  McMullen,  1976). 

This  study  will  determine  validity  of  the  Attack  Pilot  Candidate  Evaluation 
Form  in  predicting  the  flight  and  gunnery  transition  grades  of  AH-1  students. 
An  additional  variable  of  interest  was  the  relationship  between  length  of 
rater-ratee  acquaintance  and  magnitude  of  the  ratings  (Freeberg,  1969; 

Lewin  and  Zwany,  1976).  i 


METHOD 


Ratees:  The  ratees  were  43  FORSCOM  aviators,  all  rotary  wing  qualified 
and  assigned  to  AH-1  transition  training  at  Fort  Rucker.  The  ratees  were 


selected  from  AH-1  class  rosters  if  their  unit  of  origin  was  one  with 
AH— 1  aircraft  in  the  TOE.  The  units  were  selected  on  a  worldwide  basis 
and  are  representative  of  aviation  units  with  COBRA  pilots. 


Raters :  The  raters  were  AH-1  qualified  aviators  from  the  units  of  the 
AH-1  students.  The  number  of  raters  in  the  sample  units  varied  considerably. 
Because  of  the  requirements  of  field  duty,  not  all  AH-1  qualified  aviators 
were  available  to  evaluate  the  students  from  their  units.  However,  no 
systematic  basis  for  nonavailability  which  would  influence  the  results  of 
this  study  was  apparent. 

Procedure :  The  AH-1  transition  course  lasts  5  weeks,  and  the  classes  are 
begun  every  two  weeks.  Beginning  in  Oct  76  when  rosters  became  available 
for  an  incoming  class  they  were  examined  and  students  arriving  from  units 
which  were  likely  to  have  an  attack  pilot  element  were  earmarked.  The 
student's  unit  was  then  contacted  to  confirm  that  a  number  of  COBRA  pilots 
were  available.  Next  a  package  of  rating  forms  was  sent  to  a  point  of 
contact  (POC)  such  as  unit  XO  or  a  senior  attack  pilot.  The  POC  then 
distributed  the  rating  forms  and  an  envelope  to  all  the  available  AH-1 
pilots  and  later  collected  them  in  sealed  envelopes  to  insure  confidentiality. 
Finally,  the  set  of  rating  forms  was  returned  to  ARI  in  a  mailer  provided  for 
that  purpose.  This  procedure  was  followed  for  all  classes  during  a  14 
month  period  between  October  1976  to  December  1977.  It  was  necessary  to 
include  this  large  number  of  classes  because  only  a  minority  of  AH-1 
students  met  the  criteria  established.  Many  of  the  students  who  could  not 
be  used  were  turnaround  Initial  Entry  Rotary  Wing  (IERW)  students  who  had 
just  finished  flight  school.  Another  large  group  came  from  units  with  no 
COBRA  pilots. 

Rating  Scale:  The  rating  form  used  was  designed  to  have  raters  discriminate 
among  ratees  on  a  set  of  desirable  characteristics  for  attack  pilots.  The 
characteristics  rated  were  identified  during  structured  interviews  of  58 
attack  pilots  with  combat  experience  at  Ft  Knox,  Ft  Hood  and  Ft  Rucker.  On 
the  evaluation  form  the  rater  (the  AH-1  qualified  pilot)  is  instructed  to 
consider  the  set  of  attack  pilot  characteristics  and  to  assign  the  AH-1 
student  a  numerical  rank,  between  1  and  25,  representing  standing  within  a 
typical  group  of  25  pilots.  The  rater  was  also  provided  space  within  which 
to  write  a  2  -  3  sentence  word  picture  justifying  the  numerical  rating 
assigned.  Additional  information  was  also  recorded  on  where  the  rating  was 
conducted  and  the  type  and  duration  of  the  relationship  between  the  rater  and 
ratee.  Detailed  instructions  were  provided,  some  of  which  only  apply  when 
the  rating  form  is  to  be  used  to  rate  a  group  of  AH-1  candidates  (see  Appendix  A). 

RESULTS  AND  DISCUSSION 

The  median  rank  order  rating  was  computed  for  each  student  from  the  set  of 
ratings  received  from  his  unit.  This  measure  was  used  to  predict  two 
criteria:  (1)  AH-1  flight  transition  grades,  and  (2)  AH-1  gunnery  grades. 

The  predictive  validity  of  the  median  rating  was  determined  by  computing 
a  Pearson's  r  between  the  predictor  and  each  criterion  grade.  The  results 
in  Table  1  show  that  the  validity  coefficient  for  ratings  on  flight  transition 
grades,  r  =  .32,  was  high  enough  to  be  useful  as  well  as  statistically 


2 


significant  (pc.Ol).  By  contrast,  the  lower  predictive  validity  of  ratings 
for  the  gunnery  phase  of  AH-1  transition  is  probably  not  useful  as  a 


TABLE  1 

CORRELATIONS  BETWEEN  TRANSITION  GRADES,  GUNNERY  GRADES  AND 
THE  MEDIAN  RATINGS  RECEIVED  BY  AH-1  STUDENTS  (N  =  45) 


Variables 

r 

P 

r2 

AH-1 

transition  and  median  rating 

.32 

<.01 

.10 

AH-1 

gunnery  and  median  rating 

.21 

A 

O 

.04 

AH-1 

transition  and  gunnery 

.33 

<.01 

predictor,  r  =  .21.  The  significant  difference  between  these  two  validities 
(p<.01)  may  be  attributable  to  differences  in  the  quality  of  grading  the 
two  phases.  During  the  flight  transition  phase,  performance  criteria  and 
IP  standardization  have  been  established  for  grading  AH-1  students.  During 
the  gunnery  stage,  grading  is  not  based  on  specified  performance  criteria, 
e.g.,  accuracy  is  not  graded.  Improvements  in  gunnery  grading  criteria  are 
needed  before  this  training  performance  can  be  adequately  predicted. 

Although  the  validities  obtained  are  not  very  high,  the  predictive  validity 
of  .32  accounts  for  more  than  10%  of  the  variance  in  transition  grades  and 
will  be  useful  in  selecting  AH-1  students.  Moreover,  the  validities  reported 
are  a  very  conservative  underestimate  of  those  which  would  be  obtained 
with  an  unrestricted  population  of  AH-1  candidates.  Because  the  ratees 
had  already  been  selected  for  AH-1  transition,  it  is  reasonable  to  expect 
that  the  ratings  of  marginal  and  average  aviators  were  somewhat  inflated. 

This  was  supported  by  positive  skewing  of  the  distribution  of  ratings  which 
suggested  the  use  of  the  median  as  a  datum.  Because  these  data  were 
obtained  by  mail,  the  number  of  ratees  was  probably  fewer  than  would  be 
possible  than  if  ratings  had  been  conducted  as  a  unit  level  operational 
procedure. 

The  criteria  grades  for  both  the  transition  and  gunnery  phases  are  not 
very  discriminating  of  training  performance  because  of  management  and  grading 
policies/practices  which  preclude  failures  and  encourage  giving  85s  to 
graduate  aviators  in  advanced  training.  Some  indication  of  this  is  provided 
by  the  means  and  standard  deviations  of  flight  transition  and  gunnery  grades 
shown  in  Table  2.  Considering  these  factors,  the  .32  validity  obtained  for 
prediction  of  gunnery  grades  is  an  encouraging  finding  in  conjunction  with  the 


high  reliability  demonstrated  by  aviator  associate  ratings  (Eastman  and 
McMullen).  Properly  used  at  the  unit  level,  associate  ratings  would  provide 
a  useful  selection  tool  to  unit  commander  and  training  officers. 


TABLE  2 

MEANS  AND 

STANDARD  DEVIATIONS  OF  AH-1  TRANSITION 
AND  GUNNERY  GRADE  (N  =  45) 

Phase  of  Instruction 

X 

SD 

Flight  Transition 

84.13 

3.04 

Gunnery 

85.93 

1.77 

No  significant  relationship  was  found  (r  =  .09)  between  the  length  of 
acquaintanceship  of  the  rater  and  ratee  and  the  magnitude  of  the  ratings 
given. 

A  related  AH-1  Candidate  Selection  Study  included  an  open  ended  section 
in  which  the  rater  gave  a  verbal  picture  of  the  ratees.  The  verbal  content 
of  this  section  was  analyzed  for  those  aviators  who  scored  above  average  in 
the  AH-1  transition.  The  comments  for  those  who  were  rated  high  (above  8.0), 
or  medium  (8-15),  and  low  (16-25)  are  presented  in  Table  3. 

CONCLUSIONS 

The  validity  (r  -  .32)  of  ratings  in  predicting  AH-1  flight  transition 
training  grades  indicates  that  ratings  of  potential  transition  students 
by  COBRA  pilots  would  provide  useful  information  to  unit  commanders 
and  training  officers  in  selecting  aviators  for  training.  The  true 
validity  of  ratings  is  anticipated  to  be  somewhat  higher  than  that  obtained 
in  this  study,  because  of  limitations  imposed  by  the  procedures  and 
available  sample. 

Highly  rated  good  students  were  regarded  to  be  aggressive  leaders  while 
the  low  rated  poor  students  lacked  aggressiveness  and  did  not  desire 
gunship  duties.  However,  factors  such  as  dependability  and  team  performance 
emphasized  by  raters  appear  to  contradict  the  self  reported  impulsive/ 
independence  of  the  ACE  group.  The  rater  received  a  questionnaire  to  rate 
the  student  identical  to  the  one  shown  in  Appendix  A. 


TABLE  3 
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fRBCSDUO  PAOE  BLANK-HOT  FILMED 


APPENDIX  A  TANK  CREW  STABILITY  QUESTIONNAIRE  (PT  5188) 


Complete  this  for  attack  pilot  candidate  evaluation 

Complete  this  form  only  if  you  are  AH-1G  qualified. 


Instructions: 

1.  Evaluate  this  man  in  your  unit/class  in  terms  of  vour  estimate  of  his 
potential  ability  to  become  a  successful  gunshi p/attack  pilot.  Determine 
where  you  think  he  would  rank  in  a  typical  group  of  25  pilots  (number  1 
the  highest  ranking,  25  the  lowest  ranking).  Consider  the  ATTACK  PILOT 
CHARACTERISTICS  below  prior  to  rating  each  man.  Consider  the  enti re 
group  you  are  asked  to  evaluate  and  the  following  restrictions  before 
beginning,  (a)  No  more  than  two  individuals  may  be  placed  in  1-5  column. 

(b)  no  two  individuals  will  be  assigned  the  same  rating  number.  Do  not 
rate  yourself. 

2.  Under  REMARKS,  write  a  2-3  sentence  word  picture  to  justify  the  numerical 
rating  you  assigned.  State  briefly  the  characteristics  (desirable  or  un¬ 
desirable)  of  this  man  that  impressed  you  most. 

3.  Your  ratings  will  remain  anonymous.  The  packet  you  picked  up  has  an 
ID  number  only  to  insure  that  you  followed  the  restrictions  when  rating. 

EVALUATED  INDIVIDUAL'S  NAME  (Last,  first)  DATE 

DAY  MONTH  YEAR 

_ , _ _ _ L _ L _ 

ATTACK  PILOT  CHARACTERISTICS 


DESIRES  GUNSHIP  DUTIES  AGGRESSIVENESS  CONFIDENCE 

TACTICAL  KNOWLEDGE  SELF-DISCIPLINE  TEAMWORK 


TIMELINESS  OF  ACTION  DRIVE 


INITIATIVE 


MECHANICAL  ABILITY  EFFECTIVE  MAP  USE  DEPENDABILITY 


CANDIDATE'S 

STANDING  WITHIN  A 

PRESENT  LOCATION 

TRANSITION 

25 

-MAN 

GROUP 

(Circle  one) 

IERW 

UNIT 

TRAINING 

( Ci rcle 

one) 

RELATIONSHIP  TO 
CANDIDATE 

HIS 

IN  SAME 

1 

6 

11 

16 

21 

(Circle  one) 

CO 

IP 

UNIT 

2 

7 

12 

17 

22 

REMARKS: 

3 

8 

13 

18 

23 

4 

9 

14 

19 

24 

5 

10 

15 

_ 

HOW  I  PNG  HAVE  YOU  KNOWN  THE  INDIVIDUAL?  YEARS  MONTHS 

RATER  ID  ? 


